Modelling and Mining of Networked Information Spaces

The Web as a text corpus Pages close in word vector space tend to be related Cluster hypothesis (van Rijsbergen 1979) The WebCrawler (Pinkerton 1994) The whole first generation of search engines weapons mass destruction p 1 p 2 Enter the Web's link structure Broder & al. 2000 p(i) = α N + (1 − α) j:j→i p(j) Text Links Meaning Connection between semantic topology (topicality or relevance) and link topology (hypertext) G = Pr[rel(p)] ~ fraction of relevant pages (generality) R = Pr[rel(p) | rel(q) AND link(q,p)] Related nodes are " clustered " if R > G (modularity) Necessary and sufficient condition for a random crawler to find pages related to start points • Stationary hit rate for a random crawler: Link-cluster conjecture η(t + 1) = η(t) ⋅ R + (1 − η(t))⋅ G ≥ η(t) η t →∞ Value added Conjecture Pages that link to each other tend to be related Preservation of semantics (meaning) A.k.a. topic drift Link-cluster conjecture L(q,δ) ≡ path(q, p) { p: path(q, p) ≤δ } ∑ { p : path(q, p) ≤ δ} € R(q,δ) G(q) ≡ Pr rel(p) | rel(q) ∧ path(q, p) ≤ δ [ ] Pr[rel(p)] JASIST 2004 10 Correlation of lexical and linkage topology L(δ): average link distance S(δ): average similarity to start (topic) page from pages up to distance δ Correlation ρ(L,S) = –0.76 The " link-content " conjecture € S(q,δ) ≡ sim(q, p) { p: path(q, p) ≤δ } ∑ { p : path(q, p) ≤ δ} Heterogeneity of link-content correlation € S = c + (1− c)e aL b edu net gov com signif. diff. a only (p<0.05) signif. diff. a & b (p<0.05) org Discussion Topic drift: Myth or reality? Mapping the relationship between links, content, and semantic topologies • Given any pair of pages, need 'similarity' or 'proximity' metric for each topology: – Content: textual/lexical (cosine) similarity – Link: co-citation/bibliographic coupling – Semantic: relatedness inferred from manual classification • Data: Open Directory Project (dmoz.org) – ~ 1 M pages after cleanup – ~ 1.3*10 12 page pairs! Content similarity € σ c p 1 , p 2 () = p 1 ⋅ p 2 p 1 ⋅ p 2 term i term j term k p 1 p 2 p 1 p 2 € σ l (p 1 , p 2) = U p 1 ∩ U p 2 U p 1 ∪ …